Focused on the problem of traditional object recognition algorithm that the artificially designed features were more susceptible to diversity of object shapes, illumination and background, a deep convolutional neural network algorithm was proposed for object recognition. Firstly, this algorithm was trained with NYU Depth V2 dataset, and single depth information was transformed into three channels. Then color images and transformed depth images in the training set were used to fine-tune two deep convolutional neural networks, respectively. Next, color and depth image features were extracted from the first fully connected layers of the two trained models, and the two features from the resampling training set were combined to train a Linear Support Vector Machine (LinSVM) classifier. Finally, the proposed object recognition algorithm was used to extract super-pixel features in scene understanding task. The proposed method can achieve a classification accuracy of 91.4% on the test set which is 4.1 percentage points higher than SAE-RNN (Sparse Auto-Encoder with the Recursive Neural Networks). The experimental results show that the proposed method is effective in extracting color and depth image features, and can effectively improve classification accuracy.